Image processing device and method

ABSTRACT

Disclosed herein is an image processing device including a background model creation section adapted to create a plurality of background models based on statistical distribution of a plurality of image frame sets, the image frame sets differing in length of time, and each of the image frame sets including a plurality of input frames; and a background image creation section adapted to create a background image by referring to the plurality of background models.

BACKGROUND

The present disclosure relates to an image processing device and methodand, more particularly, to an image processing device and method thatallow for stable creation of a background while adjusting to backgroundchanges.

A process for extracting an object that does not exist in a backgroundimage built in advance by comparing an input image supplied from acamera or other device and the background image is generally referred toas a background subtraction algorithm (refer, for example, to JapanesePatent Laid-Open Nos. 2009-31939, 2009-265827 and, 2009-69996).

In a background subtraction algorithm, the following hypotheses are madewith respect to a background image depending on the situation to whichthe algorithm is applied:

-   -   Background values basically remain unchanged.    -   Changes in the background values fall within a given range.    -   Changes in the background values fall within a plurality of        given ranges.    -   Changes in the background values can be predicted in one way or        the other.

A background model statistically describes these hypotheses. Besides, abackground image is obtained by processing background models in one wayor the other and converting the background models into a form that canbe compared with an input image (normal image data format).

Statistical distribution of a group of past input images is used tobuild background models. Therefore, it is preferred that the group ofinput images should be observed over a long period of time to stablydistinguish between background and non-background.

Besides, when a practical usage environment is assumed, changes in thefollowing are highly possible with respect to the above hypotheses:

-   -   Changes in lighting conditions (sunshine and indoor lighting)    -   Changes in background objects (e.g., a car enters an image frame        and comes to a stop)    -   Changes in camera orientation

As a result, countermeasures against such changes are necessary.

SUMMARY

However, observation of a group of input images over a long period oftime to stably distinguish between background and non-background leadsto a contradiction which is that it is difficult to adjust to suchbackground changes.

That is, it has been difficult to strike a balance between twocontradicting objects:

-   -   Stably distinguish between background and non-background    -   Quickly adjust to background changes

The present disclosure has been devised in light of the foregoing, andit is desirable to create a stable background while adjusting tobackground changes.

An image processing device according to an embodiment of the presenttechnology includes a background model creation section and a backgroundimage creation section. The background model creation section creates aplurality of background models based on statistical distribution of aplurality of image frame sets. The image frame sets differ in length oftime, and each of the image frame sets includes a plurality of inputframes. The background image creation section creates a background imageby referring to the plurality of background models.

An image processing method according to another embodiment of thepresent technology is used by an image processing device. The imageprocessing method creates a plurality of background models based onstatistical distribution of a plurality of image frame sets. The imageframe sets differ in length of time, and each of the image frame setsincludes a plurality of input frames. The image processing method alsocreates a background image by referring to the plurality of backgroundmodels.

In the embodiment of the present technology, a plurality of backgroundmodels based on statistical distribution of a plurality of image framesets are created. The image frame sets differ in length of time, andeach of the image frame sets includes a plurality of input frames. Then,a background image is created by referring to the plurality ofbackground models.

The present technology allows for creation of a stable background whileadjusting to background changes.

It should be noted that the effect described in the presentspecification is merely illustrative, that the effect of the presenttechnology is not limited to that described in the presentspecification, and that there may be an additional effect.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a configuration example of animage processing device to which the present technology is applied;

FIG. 2 is a diagram describing a background model creation method;

FIG. 3 is a flowchart describing image processing handled by the imageprocessing device;

FIG. 4 is a flowchart describing a background image creation process instep S104 of FIG. 3; and

FIG. 5 is a block diagram illustrating a configuration example of apersonal computer.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

A description will be given below of a mode for carrying out the presentdisclosure (hereinafter referred to as an embodiment).

Configuration Example of the Image Processing Device

FIG. 1 is a block diagram illustrating a configuration example of animage processing device to which the present technology is applied. Theexample of FIG. 1 illustrates an image processing device which employs abackground subtraction algorithm, a process that extracts an object thatdoes not exist in a background image built in advance. Such extractionof an object is achieved by comparing an input image supplied from acamera or other device and the background image.

That is, an image processing device 100 illustrated in FIG. 1 creates abackground model from an input image, creates a background image, andfinds a difference in background as image processing.

In the example of FIG. 1, the image processing device 100 includes animage frame set updating section 101, an image frame set storage section102, a background model creation section 103, a background model storagesection 104, a background image creation section 105, and a differencecalculation/threshold processing section 106.

The image frame set updating section 101 updates “a plurality of imageframe sets that differ in length of time” stored in the image frame setstorage section 102 each time an input image is updated. Morespecifically, the image frame set updating section 101 adds an inputimage to each image frame set each time a maximum number of frames ofthe input image specified for each image frame set (e.g., 100 frames forthe first set and 1000 frames for the second set) are updated. Further,each time each image frame set exceeds its maximum frame count, theimage frame set updating section 101 discards the oldest image frame inthe set.

It should be noted that the following is also acceptable. That is, theimage frame set updating section 101 has a “thinning interval” N_i(where i is one of a plurality of image frame sets) for each set. For anith image frame set, the image frame set updating section 101 adds aninput image to the set and discards the oldest frame in the set eachtime N_i frames of the input image are updated. Assuming, for example,that there are two image frame sets as a whole, and that a thinninginterval N_1 for the first set is 1 and a thinning interval N_2 for thesecond set is 10, the second set holds image frames that have aten-times longer time scale than those of the first set.

The image frame set storage section 102 stores a plurality of imageframe sets that differ in length of time. It should be noted that theimage frame sets may be sets with some input frames thinned outtherefrom as described above. Further, the image frame sets may have thesame or different first frame shooting times. That is, they may havedifferent time axes. Still further, the plurality of image frame setsmay have non-overlapping durations of shooting. That is, it is notnecessary for the shooting durations of the plurality of image framesets to match perfectly, and a partial match is sufficient.

The background model creation section 103 creates a background model byanalyzing statistical distribution for each of the image frame setsstored in the image frame set storage section 102. More specifically,the background model creation section 103 creates a pixel valuehistogram for each pixel position as illustrated in FIG. 2 for eachimage frame set. As a result, a histogram set is obtained by observingeach pixel position over a different length of time.

Although, in the above description, pixel values of image frame sets aredirectly grouped into histograms, the histograms may be approximated byappropriate mathematical formulas (e.g., high-degree polynomials).Alternatively, each histogram may be represented by a mixed Gaussiandistribution.

Further, as another method for creating background histograms (methodfor storing past tendencies in a more pronounced manner), one frame-oldhistograms and histograms created at the current time by the abovemethod from image frame sets may be mixed at a given ratio.

The background model storage section 104 stores, as a background model,a set of histograms obtained by observing each pixel location over adifferent length of time.

The background image creation section 105 creates a background imagefrom background histograms as background models of the background modelstorage section 104. More specifically, the background image creationsection 105 determines pixel values of a background image from theplurality of histograms obtained by observing each pixel over adifferent length of time. In the case of a background, the pixel valuesare approximately constant. Therefore, a clear peak (i.e., most frequentposition (pixel value)) is formed in every histogram. As a result, abackground value is fixed by determining whether or not there is anyclear peak in a histogram. It should be noted that when a maximum value,i.e., a top portion, has sharpness higher than a given threshold, thisvalue should preferably be considered a peak.

A description will be given below using two kinds of (long- andshort-term) histograms having different lengths of time for simplicity.It should be noted that more than two kinds of histograms may also beused.

The background image creation section 105 determines whether or not eachof the long- and short-term histograms has a peak and determines thepeak of either of the two histograms as a background value. Morespecifically, the long-term histogram is used preferentially over theshort-term one to determine a background value. It should be noted,however, that a background value is determined by using the short-termhistogram only if the long-term histogram peak is ambiguous and if theshort-term histogram peak is sharp. As a result, a background image iscreated.

The difference calculation/threshold processing section 106 compares thebackground image created by the background image creation section 105and the input image of the current frame, thus separating background andnon-background areas. More specifically, when an absolute value of adifference between pixel values is greater than a predetermined giventhreshold, this area is considered a “non-background area.” Otherwise(when the absolute value is equal to or smaller than the threshold),this area is considered a “background area.” Alternatively, a“normalized cross-correlation” of a pixel patch centering around a pixelposition of interest may be calculated and compared against a threshold.

Processing Example of the Image Processing Device

A description will be given next of image processing performed by theimage processing device 100 with reference to the flowchart illustratedin FIG. 3.

In step S101, the image frame set updating section 101 loads an inputimage. In step S102, the image frame set updating section 101 updates aplurality of image frame sets stored in the image frame set storagesection 102 that differ in length of time.

In step S103, the background model creation section 103 createsbackground models by analyzing a statistical distribution for each ofthe image frame sets stored in the image frame set storage section 102.

In step S104, the background image creation section 105 creates abackground image from background histograms as background models of thebackground model storage section 104. Although details of thisbackground image creation process will be described later with referenceto FIG. 4, pixel values of the background image are determined from aplurality of histograms that are obtained by observing each pixel over adifferent length of time to create the background image in step S104.

In step S105, the difference calculation/threshold processing section106 calculates difference and processes a threshold. That is, thedifference calculation/threshold processing section 106 compares thebackground image created in step S104 and the input image of the currentframe, thus separating background and non-background areas and creatinga background difference image.

A background difference image is created as described above.

A description will be given next of the background image creationprocess in step S104 of FIG. 3 with reference to FIG. 4. It should benoted that this process is performed for each pixel.

In step S131, the background image creation section 105 determineswhether or not there is a peak in the short-term histograms (hereinafterdesignated as H0) of the plurality of histograms obtained by observationover different lengths of time. When the background image creationsection 105 determines in step S131 that there is a peak in theshort-term histograms (H0), the process proceeds to step S132.

In step S132, the background image creation section 105 determineswhether or not there is a peak in the long-term histograms (hereinafterdesignated as H1). If the background image creation section 105determines in step S132 that there is no peak in the long-termhistograms (H1), the process proceeds to step S133. In step S133, thebackground image creation section 105 creates a background image pixelby using the peak H0 of the short-term histograms as a background pixelvalue.

When the background image creation section 105 determines in step S132that there is a peak in the long-term histograms (H1), the processproceeds to step S134. In step S134, the background image creationsection 105 determines whether or not the short-term histogram peak H0and the long-term histogram peak H1 have the same pixel value.

When the background image creation section 105 determines in step S134that the short-term histogram peak H0 and the long-term histogram peakH1 have the same pixel value, the process proceeds to step S135. In stepS135, the background image creation section 105 creates a backgroundimage pixel by using the short-term histogram peak HO (i.e., peak H1 ofthe long-term histograms) as a background pixel value.

On the other hand, if the background image creation section 105determines in step S131 that there is no peak in the short-termhistograms (H0), the background image creation section 105 determinesthat it is not a background region but a player region and the processproceeds to step S136. In step S136, the background image creationsection 105 determines whether or not there is a peak in the long-termhistograms (H1). When the background image creation section 105determines in step S136 that there is a peak in the long-term histograms(H1), the process proceeds to step S137.

Further, if the background image creation section 105 determines in stepS134 that the short-term histogram peak H0 and the long-term histogrampeak H1 do not have the same pixel value, the process also proceeds tostep S137.

In step S137, the background image creation section 105 creates abackground image pixel by using the long-term histogram peak H1 as abackground pixel value.

If the background image creation section 105 determines in step S136that there is no peak in the long-term histograms (H1), the processproceeds to step S138. In step S138, the background image creationsection 105 determines that the peak is unknown because of the lack of apeak in both of the histograms. However, if either of the histograms isselected, the background image creation section 105 creates a backgroundimage pixel by using, for example, the long-term histogram peak H1 as abackground pixel value.

After each of steps S133, S135, S137, and S138, the process returns tostep S104 in FIG. 3 and then proceeds to step S105.

As described above, preference is given to long-term histograms overshort-term ones to determine a background value when the backgroundimage is created. A background value is determined by using theshort-term histogram only if the long-term histogram peak is ambiguousand if the short-term histogram peak is sharp.

Thus, the present technology has a plurality of background models basedon statistical distribution of a plurality of image frame sets thatdiffer in length of time as a background subtraction algorithm.Background models are built by referring to the plurality of backgroundmodels.

This ensures improvement in two contradicting characteristics, namely,stable capability to distinguish between background and non-background,a feature obtained by observation of a group of input images over a longperiod of time and capability to quickly adjust to background changes, afeature obtained by observation of a group of input images over a shortperiod of time.

Thus, the present technology allows for stable creation of a backgroundwhile adjusting to background changes.

Personal Computer

The above series of processes may be performed by software or hardware.If the series of processes are performed by software, a program makingup the software is installed to a computer. Here, the computer includesnot only computers that are built into dedicated hardware but alsogeneral-purpose personal computers capable of executing a variety offunctions as a result of installation of a variety of programs.

FIG. 5 is a block diagram illustrating a hardware configuration exampleof a personal computer for executing the above series of processes byusing a program.

In a personal computer 500, a CPU (Central Processing Unit) 501, a ROM(Read Only Memory) 502, a RAM (Random Access Memory) 503 are connectedto each other by a bus 504.

An I/O (Input/Output) interface 505 is further connected to the bus 504.An input section 506, an output section 507, a storage section 508, acommunication section 509, and a drive 510, are connected to the I/Ointerface 505.

The input section 506 includes a keyboard, a mouse, a microphone, and soon. The output section 507 includes a display, a speaker, and so on. Thestorage section 508 includes a harddisk, a non-volatile memory, and soon. The communication section 509 includes a network interface and soon. The drive 510 drives a removable medium 511 such as magnetic disk,optical disc, magneto-optical disk, or semiconductor memory.

In the personal computer 500 configured as described above, the CPU 501loads a program from the storage section 508 into the RAM 503 forexecution via the I/O interface 505 and the bus 504, thus allowing theabove series of processes to be performed.

The program executed by the computer (CPU 501) can be recorded on theremovable medium 511 to be supplied. The removable medium 511 is, forexample, a package medium that includes, for example, a magnetic disk(including flexible disk), an optical disc (e.g., CD-ROM (CompactDisc-Read Only Memory), DVD (Digital Versatile Disc)), a magneto-opticaldisk, or a semiconductor memory. Further, the program can be suppliedvia a wired or wireless transmission medium such as local area network,the Internet, or digital satellite broadcasting.

The program can be installed to the storage section 508 of the computervia the I/O interface 505 by inserting the removable medium 511 into thedrive 510. Alternatively, the program can be received by thecommunication section 509 via a wired or wireless transmission mediumand installed to the storage section 508. In addition to the above, theprogram can be installed in advance to the ROM 502 or the storagesection 508.

It should be noted the program executed by the computer may perform theprocesses chronologically in accordance with a sequence described in thepresent specification. Alternatively, the program may perform theprocesses in parallel or when necessary as when the program is invoked.

Further, steps that describe the program recorded in the recordingmedium include not only the processes performed chronologically inaccordance with the described sequence but also those that are performedin parallel or individually if not necessarily performedchronologically.

In the present specification, the term “system” refers to an apparatusas a whole that includes a plurality of devices.

For example, embodiments of the present disclosure are not limited tothat described above and may be changed in various ways withoutdeparting from the scope of the present disclosure.

In the present disclosure, for example, a function can be providedthrough cloud computing in which the function is shared among aplurality of devices and handled in collaboration with each other via anetwork.

Further, components described in the above description as a singledevice (or processing section) may be separated into a plurality ofdevices (or processing sections). Conversely, components described inthe above description as a plurality of devices (or processing sections)may be combined into a single device (or processing section). Stillfurther, other component may be naturally added to the above componentsof each device (or processing section). Moreover, as long as thecomponents or operation of the system as a whole substantially remainthe same, some components of a device (or processing section) may beincluded in the components of other device (or other processingsection). That is, the present technology is not limited to theembodiment described above and may be changed in various ways withoutdeparting from the scope of the technology.

Although a detailed description has been given of a preferred embodimentof the present disclosure with reference to the accompanying drawings,the disclosure is not limited to such an example. It is obvious that aperson having ordinary knowledge in the technical field to which thepresent disclosure pertains can conceive of a variety of modificationand correction examples within the realms of the technical ideadescribed in the claims, and it is naturally understood that these alsofall within the scope of the present disclosure.

It should be noted that the present technology may also have one of thefollowing configurations:

(1) An image processing device that includes:

a background model creation section that creates a plurality ofbackground models based on statistical distribution of a plurality ofimage frame sets, the image frame sets differing in length of time, andeach of the image frame sets including a plurality of input frames; and

a background image creation section that creates a background image byreferring to the plurality of background models.

(2) The image processing device of feature (1), in which

the background image creation section creates a background image byusing pixel values of short-term background models created on the basisof statistical distribution of short-term image frame sets if there isno peak with sharpness higher than a threshold in long-term backgroundmodels created on the basis of statistical distribution of long-termimage frame sets and if there is a peak with sharpness higher than thethreshold in the short-term background models, and

otherwise, the background image creation section creates a backgroundimage by preferentially using the pixel values of the long-termbackground models.

(3) The image processing device of feature (1) or (2) further including:

a difference calculation processing section that separates a newlysupplied input frame into background and non-background areas bycomparing the newly supplied input frame and the background image.

(4) The image processing device of any one of features (1) to (3) stillfurther including:

an image frame set updating section that updates the image frame setseach time a given number of input frames are updated.

(5) The image processing device of any one of features (1) to (4), inwhich

the plurality of image frame sets are sets with some of the plurality ofinput frames thinned out therefrom.

(6) The image processing device of any one of features (1) to (5), inwhich

the plurality of image frame sets have different first frame shootingtimes.

(7) The image processing device of any one of features (1) to (6), inwhich

the plurality of image frame sets do not have overlapping durations ofshooting.

(8) The image processing device of any one of features (1) to (7), inwhich

the statistical distribution is a pixel value histogram.

(9) The image processing device of any one of features (1) to (7), inwhich

the statistical distribution is a mixed Gaussian distribution.

(10) The image processing device of any one of features (1) to (7), inwhich

the statistical distribution is a function approximation of pixel valuedistribution.

(11) An image processing method used by an image processing device, themethod including:

creating a plurality of background models based on statisticaldistribution of a plurality of image frame sets, the image frame setsdiffering in length of time, and each of the image frame sets includinga plurality of input frames; and

creating a background image by referring to the plurality of backgroundmodels.

The present disclosure contains subject matter related to that disclosedin Japanese Priority Patent Application JP 2015-227610 filed in theJapan Patent Office on Nov. 20, 2015, the entire content of which ishereby incorporated by reference.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

What is claimed is:
 1. An image processing device comprising: abackground model creation section adapted to create a plurality ofbackground models based on statistical distribution of a plurality ofimage frame sets, the image frame sets differing in length of time, andeach of the image frame sets including a plurality of input frames; anda background image creation section adapted to create a background imageby referring to the plurality of background models.
 2. The imageprocessing device of claim 1, wherein the background image creationsection creates a background image by using pixel values of short-termbackground models created on the basis of statistical distribution ofshort-term image frame sets if there is no peak with sharpness higherthan a threshold in long-term background models created on the basis ofstatistical distribution of long-term image frame sets and if there is apeak with sharpness higher than the threshold in the short-termbackground models, and otherwise, the background image creation sectioncreates a background image by preferentially using the pixel values ofthe long-term background models.
 3. The image processing device of claim1 further comprising: a difference calculation processing sectionadapted to separate a newly supplied input frame into background andnon-background areas by comparing the newly supplied input frame and thebackground image.
 4. The image processing device of claim 1 stillfurther comprising: an image frame set updating section adapted toupdate the image frame sets each time a given number of input frames areupdated.
 5. The image processing device of claim 1, wherein theplurality of image frame sets are sets with some of the plurality ofinput frames thinned out therefrom.
 6. The image processing device ofclaim 1, wherein the plurality of image frame sets have different firstframe shooting times.
 7. The image processing device of claim 1, whereinthe plurality of image frame sets do not have overlapping durations ofshooting.
 8. The image processing device of claim 1, wherein thestatistical distribution is a pixel value histogram.
 9. The imageprocessing device of claim 1, wherein the statistical distribution is amixed Gaussian distribution.
 10. The image processing device of claim 1,wherein the statistical distribution is a function approximation ofpixel value distribution.
 11. An image processing method used by animage processing device, the method comprising: creating a plurality ofbackground models based on statistical distribution of a plurality ofimage frame sets, the image frame sets differing in length of time, andeach of the image frame sets including a plurality of input frames; andcreating a background image by referring to the plurality of backgroundmodels.